Add EPP in-flight requests and tokens metrics (#1089) by chethanuk · Pull Request #1697 · llm-d/llm-d-router

chethanuk · 2026-06-20T19:31:44Z

Summary

Implements #1089 (xref upstream GAIE kubernetes-sigs/gateway-api-inference-extension#2072): EPP observability for in-flight requests and tokens. Adds two complementary signals:

Per-endpoint in-flight load (the core of [Observability] An EPP metric that reports inflight tokens and requests #1089) — llm_d_epp_inflight_requests and llm_d_epp_inflight_tokens, gauges labelled endpoint_name,namespace,producer_name, sourced from the InFlightLoadProducer's live per-endpoint request/token trackers via a custom prometheus.Collector (mirrors pkg/epp/metrics/collectors/inference_pool.go, the GAIE per-pod-queue-size pattern). This answers "which replica is loaded", matching docs/configs: replace deprecated maxPrefixBlocksToMatch and blockSize with maxPrefixTokensToMatch and blockSizeTokens #2072's per-pod intent.
Per-model in-flight requests — llm_d_epp_request_inflight, a sibling of request_running labelled model_name,target_model_name,fairness_id,priority, capturing the full admitted->completed window inside the EPP.

What

Per-endpoint collector (inflight_requests / inflight_tokens):

New pkg/epp/metrics/collectors/inflight_load.go — Collect emits one const gauge per endpoint from requestTracker.snapshot() / tokenTracker.snapshot(); no double-counting, cleanup follows the producer's endpoint lifecycle.
tokens = uncached prompt tokens, optionally plus estimated output when the producer's addEstimatedOutputTokens is set.
The InFlightLoadProducer self-registers the collector at construction (no startup-ordering hook); it no-ops when the plugin is not configured.
producer_name label keeps series distinct and collision-free when multiple producers are configured.

Per-model gauge (request_inflight):

Incremented at the admission boundary (right after Director.HandleRequest succeeds, where model/priority/fairness labels are populated).
Decremented in the Process defer, guaranteed on error/disconnect/panic paths (guarded by RequestContext.RequestInflight).
New-only metric (no deprecated legacy twin), following the request_ttft_seconds pattern.

Why

request_running is incremented only at dispatch, so it misses the in-EPP parse/admit/schedule window; the per-model request_inflight captures the full admitted->completed window. The per-endpoint inflight_requests/inflight_tokens expose per-replica queue depth and token pressure — the load-aware-routing signal #1089/#2072 are about — from data the InFlightLoadProducer already maintains (previously visible only via /debug/plugins/state and the token-load scorer).

Testing

make test-filter PATTERN=TestInflightRequestsMetrics TYPE=epp and the new TestInFlightLoadCollector* tests (golden output).
Full make test: epp + sidecar unit suites with -race, GAIE e2e, scheduler e2e — all pass.
make lint (0 issues), go build/go vet/gofmt clean.
Docs: docs/metrics.md documents all three metrics.

Fixes #1089

ahg-g · 2026-06-21T12:40:08Z

+limitations under the License.
+*/
+
+package collectors


The metric should be owned and updated by the inflight load producer plugin. See as an example the metrics owned and emitted by the disagg handler: pkg/epp/framework/plugins/scheduling/profilehandler/disagg/metrics.go

Implements llm-d#1089 (xref GAIE llm-d#2072): EPP in-flight observability, owned by the in-flight load producer plugin. - Per-endpoint llm_d_epp_inflight_requests / llm_d_epp_inflight_tokens gauges via a custom prometheus.Collector reading the InFlightLoadProducer snapshot. Registered through the plugin metrics recorder (handle.Metrics()) with producer_name as a constant descriptor label, so multiple producers register without collision. - Per-model llm_d_epp_request_inflight gauge owned by the producer: incremented in PreRequest, decremented exactly once via a per-request PluginState entry's OnEvicted, so it stays balanced across completion, error, abort, and reaper paths. Mirrors the metric-ownership pattern of the disagg handler (profilehandler/disagg/metrics.go). - Adds InferenceRequest.IncomingModel so the producer can label the gauge with the incoming model name. - docs/metrics.md documents all three metrics. Fixes llm-d#1089 Signed-off-by: ChethanUK <chethanuk@outlook.com>

Resolve the InFlightLoadProducerFactory conflict in inflightload's producer.go. main added output-ratio and max-estimated-output-token validation ahead of construction, while this branch switched the constructor to a named producer variable so the in-flight metrics collector can be registered against the producer before returning. Keep both behaviors: validate the config first, then build through the named variable so metric ownership stays inside the producer plugin and the new output-token caps are still enforced. Signed-off-by: ChethanUK <chethanuk@outlook.com>

chethanuk · 2026-07-13T14:32:50Z

@ahg-g / @elevran / @shmuelk Please review and merge this, or let me know if any changes are needed. Since main keeps updating, it would be great to merge this soon. I have completed work on other PRs and would like to raise them as well..

elevran · 2026-07-14T10:52:45Z

nit: these two wording changes are unrelated to the rest of the PR.

elevran · 2026-07-14T10:53:51Z

Please rebase on main to pick up fixes for failing tests.
Overall - LGTM.

chethanuk · 2026-07-14T17:54:08Z

Rebased please merge

ahg-g · 2026-07-14T19:59:31Z

+
+package collectors
+
+import (


I don't believe we need this file at all

Removed. The per-endpoint metrics are now plain gauges owned by the producer in inflightload/metrics.go, no custom collector.

ahg-g · 2026-07-14T20:05:17Z

 	return p.requestTracker.get(eid)
 }

+// InFlightRequestsSnapshot returns a copy of the per-endpoint in-flight request


I am not sure we need any of this, can't we just define the metrics as guages and update them in every place were tokenCounter and requestCounter are updated?

Done. Dropped the snapshot methods and the collector; the gauges are now set from the live counter value at every point the request/token counters change (admission, release/eviction, endpoint removal deletes the series). Setting from the live value rather than mirroring inc/dec keeps the gauge correct when a late release lands on an orphaned counter after an endpoint delete/recreate, and it can never go negative.

Signed-off-by: ChethanUK <chethanuk@outlook.com>

chethanuk · 2026-07-22T13:51:34Z

@ahg-g Please review when you got time thanks

chethanuk requested review from a team and shmuelk as code owners June 20, 2026 19:31

chethanuk requested review from elevran and liu-cong June 20, 2026 19:31

github-actions Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 20, 2026

ahg-g reviewed Jun 21, 2026

View reviewed changes

chethanuk force-pushed the in-flight-requests branch from 5160bf7 to 62ee968 Compare June 21, 2026 22:16

github-actions Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 21, 2026

Merge branch 'main' into in-flight-requests

4cfd55b

chethanuk requested a review from ahg-g June 29, 2026 10:42

elevran reviewed Jul 14, 2026

View reviewed changes

Comment thread docs/operations.md

elevran Jul 14, 2026

Copy link
Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: these two wording changes are unrelated to the rest of the PR.

elevran previously approved these changes Jul 14, 2026

View reviewed changes

chethanuk added 2 commits July 14, 2026 19:38

Merge branch 'main' into in-flight-requests

299c24e

Merge branch 'main' into in-flight-requests

312c8f0

Merge branch 'main' into in-flight-requests

64db54c

ahg-g reviewed Jul 14, 2026

View reviewed changes

chethanuk added 2 commits July 17, 2026 18:34

Merge branch 'main' into in-flight-requests

5121842

Replace in-flight load collector with plugin-owned gauges

8418f8c

Signed-off-by: ChethanUK <chethanuk@outlook.com>

chethanuk dismissed elevran’s stale review via 8418f8c July 17, 2026 19:09

chethanuk requested review from ahg-g and elevran July 17, 2026 19:39

Merge branch 'main' into in-flight-requests

6bf4b8d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add EPP in-flight requests and tokens metrics (#1089)#1697

Add EPP in-flight requests and tokens metrics (#1089)#1697
chethanuk wants to merge 9 commits into
llm-d:mainfrom
chethanuk:in-flight-requests

chethanuk commented Jun 20, 2026

Uh oh!

ahg-g Jun 21, 2026

Uh oh!

chethanuk commented Jul 13, 2026

Uh oh!

elevran Jul 14, 2026

Uh oh!

elevran commented Jul 14, 2026

Uh oh!

chethanuk commented Jul 14, 2026

Uh oh!

ahg-g Jul 14, 2026

Uh oh!

chethanuk Jul 17, 2026

Uh oh!

ahg-g Jul 14, 2026

Uh oh!

chethanuk Jul 17, 2026

Uh oh!

chethanuk commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		package collectors

		import (

Uh oh!

Conversation

chethanuk commented Jun 20, 2026

Summary

What

Why

Testing

Uh oh!

ahg-g Jun 21, 2026

Choose a reason for hiding this comment

Uh oh!

chethanuk commented Jul 13, 2026

Uh oh!

elevran Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

elevran commented Jul 14, 2026

Uh oh!

chethanuk commented Jul 14, 2026

Uh oh!

ahg-g Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

chethanuk Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

ahg-g Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

chethanuk Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

chethanuk commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants